The Bucks, 11-10 before Friday night's game against the Celtics, began the season with a victory, then lost six straight. The Bucks recovered with a seven-game winning streak before losing, 119-104, to the Atlanta Hawks on Wednesday.
"Tough start, but I think everybody here's patient," Rivers said pregame. "Why would I listen to someone I'm not going to take advice from? You know what I mean by that. I think we're going to be one of the teams, but we're not yet. We're not. So, we have work to do.
"It's funny, when we go on the road we hear that, and it's laughable for us. But our record says what we are and we're not where we want to be, so we get it."
Rivers, who guided the Celtics to the 2007-08 NBA title, said the absence of Khris Middleton and off-target shooting held the Bucks back. Middleton (ankle surgery) was listed as probable for Friday's game.
"It's not like he didn't know how to play basketball, [but] as far as continuity he's not going to have that," Rivers said of Middleton. "Your key guys don't just walk on the floor, but I will say that the couple times Khris has played for me and he's missed a big chunk of the games, he's come back and he's looked amazing. Probably as well as any player I've ever seen.
"When he does come back, I'm hoping it's that Khris Middleton tonight."
Rivers is hoping to have Middleton combine with Giannis Antetokounmpo and Damian Lillard as a big three scoring force.
"Every coach, you want your key guys, and the more games you have them the better, especially when they miss training camp. They've had zero practices -- that hurts your team in the long run. So, every time we can get them in a practice, any time we can get them on the floor in a game, and him Giannis and Damian, it's going to help our team grow. That's where we're behind teams that have had their key guys together. But it'll come. It's just going to take a minute.
"The last 10 games we're in the top five on offense, top five in defense. We're making shots. Guys that can make shots are making shots. Brook [Lopez] started the season not making 3s. Brook didn't forget how to shoot. Gary Trent is one of the best 3-point shooters in the league, and at one point he was at 20 percent. We never panicked, like, of course, Gary's going to make shots at some point. We just remained patient.
"We had a bunch of them all at the same time. Trying to explain that is hard. We just weren't making shots, and now we are."
The Celtics won the first two meetings this season with the Bucks: 119-108 on Oct. 28, and 113-107 in Milwaukee Nov. 10. Friday's game was the final game between the teams this season.
"We were talking about that today," Rivers said. "It is strange. You can look at it in a positive way -- well, we don't have to play Boston the rest of the year, and if we see them again, it will be in the playoffs, and we'll be happy about that.
"It is a quirky schedule. A lot of it is trying to get the tournament [NBA Cup] to fit in games."
Last season, Payton Pritchard sparked the Celtics with 31 points off the bench in a 122-119 win over the Bucks. Asked about Pritchard, Rivers said: "I've been a fan for a while."
As for defending against him, Rivers added: "We didn't mention him. We're going to try to change our luck."